Modeling Time Dependencies in the Mixture of Experts

The Mixture of Experts, as it was originally formulated, is a static algorithm in the sense that the output of the network, and parameter updates during training, are completely independent from one time step to the next. This independence creates difficulties when the model is applied to time series prediction. We address this by adding memory to the Mixture of Experts. A Gaussian assumption on each Experts’ error is replaced by a chi-square distribution on the local (in time) root mean square error. We derive new gradient descent equations, and present a simulation that demonstrates an improvement in the segmentation of a time series over the classical algorithm.